Recently I was working with a client and I decided to do a little experiment, because sometimes I have nothing better to do. I wanted to know whether changing the crawl rate in Google’s Webmaster Tools really made a difference. Part of me felt Google just put it there to make people feel that Google will come to their site more often, but part of me wanted to trust Google.
I quickly wrote a script that would stored all the user agent info, page visited, and date visited in a database. I let it run for 200 days, 90 of which were set on the “Faster” crawl speed (the rest were on “Normal”). I also tracked the time from which a new blog was posted and when Google first visited that post. I tracked the site for 55 days on “Normal,” 90 days on “Faster,” and then the last 55 on “Normal” again.
Here is some background information before we get into the data I collected. The website had 45 pages cached in Google and was over a year old when I began this experiment. New blog posts were added between 2 and 3 times a week, adding fresh new content to the site. The site ranks well for its brand and a few other keywords related to its field.
During the course of the experiment over 80 new blog posts were added to the site. While the crawl rate was set at “Normal” it took Google an average of 3.4 days to first visit the page. When the crawl rate was set at “Faster” it took Google an average of 2.9 days to first visit the page. That was not much of an improvement in my book, but interesting nevertheless. Note: This is not the time when the page first appeared cached in the index, but when the bot first visited the page itself.
While the crawl rate was set to “Normal,” Google visited the site an average of 6 times a day and favored visiting the site on Tuesdays, Fridays, and Sundays. When the crawl rate was set to “Faster,” Google visited the site an average of 7.5 times a day and favored visiting the site on Wednesdays, Thursdays, and Saturdays. Over 90 days it visited 135 pages more than if I would have left it set to “Normal.”
If you look at the graph above you will see there is slight improvement while the crawl rate is set to “Faster,” but only for a portion of the 90 days. After seeing the results I have come to the conclusion that changing the crawl rate in Google’s Webmaster Tools makes no substantial difference to this site.
